Coherent pitch and intensity modification of speech signals

ABSTRACT

A method comprising: receiving an utterance, an original pitch contour of the utterance, and a target pitch contour for the utterance, wherein the utterance comprises a plurality of consecutive frames, and wherein at least one of said frames is a voiced frame; calculating an original intensity contour of said utterance; generating a pitch modified utterance based on the target pitch contour; calculating an intensity modification factor for each of said frames, based on said original pitch contour and on said target pitch contour, to produce a sequence of intensity modification factors corresponding to said plurality of consecutive frames; calculating a final intensity contour for said utterance by applying said intensity modification factors to said original intensity contour; and generating a coherently modified speech signal by time dependent scaling of the intensity of said pitch modified utterance according to said final intensity contour.

BACKGROUND

The present invention relates to the fields of speech synthesis andspeech processing.

Pitch modification is an important processing component of expressiveText-To-Speech (TTS) synthesis and voice transformation. The pitchmodification task may generally appear either in the context of TTSsynthesis or in the context of natural speech processing, e.g. forentertainment applications, voice disguisement applications, etc.

Applications such as affective Human Computer Interface (HCl), emotionalconversational agents and entertainment, demand for extreme pitchmodification capability which preserves speech naturalness. However, itis widely acknowledged that pitch modification and synthesized speechnaturalness are contradictory requirements.

Pitch modification may be performed, for example, over anon-parameterized speech waveform using Pitch-Synchronous Overlap andAdd (PSOLA) method or by using a parametric speech representation.Regardless of the method used, significant raising or lowering of theoriginal tone of speech segments may significantly deteriorate theperceived naturalness of the modified speech signal.

The foregoing examples of the related art and limitations relatedtherewith are intended to be illustrative and not exclusive. Otherlimitations of the related art will become apparent to those of skill inthe art upon a reading of the specification and a study of the figures.

SUMMARY

The following embodiments and aspects thereof are described andillustrated in conjunction with systems, tools and methods which aremeant to be exemplary and illustrative, not limiting in scope.

There is provided, in accordance with an embodiment, a methodcomprising: receiving an utterance, an original pitch contour of theutterance, and a target pitch contour for the utterance, wherein theutterance comprises a plurality of consecutive frames, and wherein atleast one of said frames is a voiced frame; calculating an originalintensity contour of said utterance; generating a pitch-modifiedutterance based on the target pitch contour; calculating an intensitymodification factor for each of said frames, based on said originalpitch contour and on said target pitch contour, to produce a sequence ofintensity modification factors corresponding to said plurality ofconsecutive frames; calculating a final intensity contour for saidutterance by applying said intensity modification factors to saidoriginal intensity contour; and generating a coherently-modified speechsignal by time-dependent scaling of the intensity of said pitch-modifiedutterance according to said final intensity contour.

There is provided, in accordance with another embodiment, a computerprogram product comprising a non-transitory computer-readable storagemedium having program code embodied therewith, the program codeexecutable by at least one hardware processor to: receive an utterance,an original pitch contour of the utterance, and a target pitch contourfor the utterance, wherein the utterance comprises a plurality ofconsecutive frames, and wherein at least one of said frames is a voicedframe; calculate an original intensity contour of said utterance;generate a pitch-modified utterance based on the target pitch contour;calculate an intensity modification factor for each of said frames,based on said original pitch contour and on said target pitch contour,to produce a sequence of intensity modification factors corresponding tosaid plurality of said consecutive frames; calculate a final intensitycontour for said utterance by applying said intensity modificationfactors to said original intensity contour; and generate acoherently-modified speech signal by time-dependent scaling of theintensity of said pitch-modified utterance according to said finalintensity contour.

There is provided, in accordance with a further embodiment, a systemcomprising: (i) a non-transitory storage device having stored thereoninstructions for: receiving an utterance, an original pitch contour ofthe utterance, and a target pitch contour for the utterance, wherein theutterance comprises a plurality of consecutive frames, and wherein atleast one of said frames is a voiced frame, calculating the originalintensity contour of said utterance, generating a pitch-modifiedutterance based on the target pitch contour, calculating an intensitymodification factor for each of said frames, based on said originalpitch contour and on said target pitch contour, to produce a sequence ofintensity modification factors corresponding to said plurality of saidconsecutive frames, calculating a final intensity contour for saidutterance by applying said intensity modification factors to saidoriginal intensity contour, and generating a coherently-modified speechsignal by time-dependent scaling of the intensity of said pitch-modifiedutterance according to said final intensity contour; and (ii) at leastone hardware processor configured to execute said instructions.

In some embodiments, the received utterance is natural speech, and themethod further comprises mapping each of said frames to a correspondingspeech class selected from a predefined set of speech classes.

In some embodiments, the calculating of the intensity modificationfactor for each of said frames is based on a pitch-to-intensitytransformation modeling the relationship between the instantaneous pitchfrequency and the instantaneous intensity of the utterance, and thepitch-to-intensity transformation is represented as a function of apitch frequency and a set of control parameters.

In some embodiments, each of said frames is mapped to a correspondingspeech class selected from a predefined set of speech classes, and themethod further comprises setting the values of said control parametersfor each of said frames according to its corresponding speech class.

In some embodiments, the method of further comprises offline modeling ofthe pitch to intensity relationship to receive said values for saidcontrol parameters according to said speech classes.

In some embodiments, the method further comprises setting said controlparameters to constant predefined values.

In some embodiments, the pitch-to-intensity transformation is based onlog-linear regression, and the set of control parameters comprises theslope coefficient of the regression line of the log-linear regression.

In some embodiments, the intensity modification factor is ten in thepower of the twentieth of the ratio of average empirical decibels peroctave multiplied by the extent of pitch modification expressed inoctaves.

In some embodiments, the value of the ratio of empirical decibels peroctave is set to six decibels per octave.

In some embodiments, the calculating of the intensity modificationfactor for each of said frames comprises: calculating a reference valueof the intensity corresponding to an original pitch frequency of theoriginal pitch contour for said each frame, by applying thepitch-to-intensity transformation to the original pitch frequency;calculating a reference value of the intensity corresponding to thetarget pitch frequency of the target pitch contour for each of saidframes by applying the pitch-to-intensity transformation to the targetpitch frequency; and dividing the reference value of the intensitycorresponding to the target pitch frequency by the reference value ofthe intensity corresponding to the original pitch frequency.

In some embodiments, the received utterance is natural speech, and theprogram code is further executable by said at least one hardwareprocessor to map each of said frames to a corresponding speech classselected from a predefined set of speech classes.

In some embodiments, each of said frames is mapped to a correspondingspeech class selected from a predefined set of speech classes, and theprogram code is further executable by said at least one hardwareprocessor to set the values of said control parameters for each of saidframes according to its corresponding speech class.

In some embodiments, the program code is further executable by said atleast one hardware processor to offline model the pitch to intensityrelationship to receive said values for said control parametersaccording to said speech classes.

In some embodiments, the system further comprises a database, wherein:each of said frames is mapped to a corresponding speech class selectedfrom a predefined set of speech classes, the calculating of theintensity modification factor for each of said frames is based on apitch-to-intensity transformation represented as a function of a pitchfrequency and a set of control parameters, and the database comprisesvalues of said control parameters per a speech class of said set ofspeech classes, and wherein said storage device have further storedthereon instructions for setting values for said control parameters foreach of said frames according to its corresponding speech class, whereinthe values are fetched from said database.

In addition to the exemplary aspects and embodiments described above,further aspects and embodiments will become apparent by reference to thefigures and by study of the following detailed description.

BRIEF DESCRIPTION OF THE FIGURES

Exemplary embodiments are illustrated in referenced figures. Dimensionsof components and features shown in the figures are generally chosen forconvenience and clarity of presentation and are not necessarily shown toscale. The figures are listed below.

FIG. 1 shows a linear regression graph which shows the correlationbetween pitch frequency and intensity within an acoustic cluster derivedfrom a TTS voice dataset;

FIG. 2A shows a flowchart of a method, constructed and operative inaccordance with an embodiment of the disclosed technique;

FIG. 2B shows a flowchart of an exemplary method, constructed andoperative in accordance with another embodiment of the disclosedtechnique; and

FIG. 3 shows an exemplary system according to an embodiment.

DETAILED DESCRIPTION

Disclosed herein is a coherent modification of pitch contour andintensity contour of a speech signal. The disclosed pitch contour andintensity contour modification may enhance pitch-modified speech signalsby improving their naturalness. The disclosed enhancement may beapplicable to virtually any type of pitch modification technique.

Some prior works, unrelated to pitch modification, reported experimentalevidence of a salient positive correlation between instantaneousfundamental frequency and instantaneous loudness of speech. For example,in P. Gramming, et al, “Relationship between changes in voice pitch andloudness”, Journal of Voice, Vol 2, Issues 2, 1988, pp. 118-126,Elsevier, 1988, that correlation phenomenon was observed in professionalsingers, healthy non-singers and people suffering from voice disorders.In Rosenberg, and J. Hirshberg, “On the correlation between energy andpitch accent in read English speech”, In Proc. Interspeech 2006,Pittsburgh, Pa., USA, September 2006, a method was proposed for pitchaccent prediction based on the pitch-energy correlation.

Advantageously, the disclosed pitch contour and intensity contourmodification may harness that correlation phenomenon to improve thesound naturalness of a speech signal otherwise damaged by pitchmodification. A statistically-proven observation of a positivecorrelation between instantaneous pitch frequency and instantaneousintensity (also “loudness”) of a speech signal is herein provided, whichis the product of experimentation performed by the inventors. Astatistical exemplification of this pitch-intensity relation is shown inFIG. 1.

The disclosed pitch contour and intensity contour modificationformulates this proven pitch-intensity interrelation and manipulates itto improve the sound naturalness of a pitch-modified speech signal. Morespecifically, the intensity of a speech segment (represented, forexample, as a frame of the speech signal) may be modified in agreementwith the pitch modification within the segment. Such coherentmodification of the pitch and intensity may significantly reduce thenaturalness loss following pitch modification.

The term “utterance”, as referred to herein, may relate tonon-synthesized utterance (i.e., natural speech produced by a livingbeing such as humans) and/or synthesized utterance (i.e.,artificially-produced speech, e.g., as a result of TTS synthesis).

The utterance may be represented as a digitized speech signal(hereinafter also referred to as ‘raw speech signal’ or simply ‘speechsignal’) and/or as a parameterized signal, as discussed below.

The speech signal may be mathematically represented as a function s(n)of discrete time instant n=0, 1, 2, . . . , N corresponding to timemoments t_(n)=n·τ, n=0, 1, 2, . . . , N where τ is the time samplinginterval, e.g. τ=1/22050 seconds (s) for a 22050 Hertz (Hz) samplingrate.

For the purpose of speech processing, the time axis may be divided toframes centered at equidistant time offsets k·Δ, where k=0, 1, 2, . . ., K is the frame index and Δ is frame size. For example, frame size Δ=5milliseconds (ms) may be used for speech signals sampled at 22050 Hz.

Hereinafter various features of the speech signal calculated at framelevel may be considered, such as pitch frequency (or simply ‘pitch’),intensity, line spectrum and spectral envelope. The calculation at framelevel may mean that a feature for frame k is derived from a portion ofthe signal enclosed within a short time window, such as, for example, 5ms, 10 ms or 20 ms, surrounding the frame center t_(k)=k·Δ. These framelevel values may be also referred to as instantaneous values at thecenters of the respective frames.

The frame centers may be expressed in discrete time instants n_(k), k=0,1, . . . , K. To this end, the frame center moments t_(k)=k·Δ expressedin seconds may be divided by the sampling interval τ and rounded to thenearest integer values. For example, if the frame size is 5 ms and thesampling rate is 22050 Hz, then the frame centers in discrete timeinstants are: n₀=0, n₁=110, n₂=220, n₃=330 etc.

The term “pitch contour”, as referred to herein, may relate to thesequence of fundamental frequency (or pitch frequency) values associatedwith the respective frame centers and may be denoted: {F0 _(k), k=0, 1,. . . , K}. The value F0 _(k)=F0(n_(k)) may represent the instantaneoustone level of speech at the time moment k·Δ. The pitch frequency may beset to zero for unvoiced frames, i.e. frames which represent aperiodicparts of the utterance where pitch is undefined.

Speech signal intensity may be a measure of the loudness. Theinstantaneous intensity may be estimated as the square root of thesignal energy. The signal energy may be measured as the sum of squaredvalues of the signal components within a short window surrounding thetime moment of interest. Equivalently, the signal energy may be measuredas the sum of the squared magnitudes of the Short-Time Fourier Transform(STFT) of the signal. The sequence {I(n_(k)), k=0, 1, . . . , K} of theinstantaneous intensity values associated with the frame centers mayform what is referred to herein as the “intensity contour”.

As an alternative to representing the utterance as a speech signal, itmay be represented parametrically as a sequence {P_(k), k=0, 1, . . . ,K} of frame-wise sets of vocoder parameters, wherein a set P_(k) may beassociated with the center n_(k) of k-th frame, P_(k)=P(n_(k)). A speechsignal corresponding to the utterance may be then reconstructed from theparametric representation (i.e., the parameterized signal):

Ω:{P(n _(k)),k=0,1, . . . ,K}

{s(n),n=0,1, . . . ,N}

The contents of the vocoder parameter set P_(k) and the reconstructionalgorithm Ω may depend on the type of the parameterization (i.e., thevocoding technique) employed. The vocoder parameter set may includespectral envelop and excitation components or Sinusoidal Modelparameters including harmonic and noise components. The pitch frequencyis generally included in the vocoder parameter set.

Frames of the speech signal may be mapped to distinct speech classes.The frame class identity labels may be hereinafter referred to as frameclassification information. A speech class may correspond to frameswhich represent certain phonetic-linguistic context. For example, theframe may belong to a certain phonetic unit which is a part of asentence subject and is preceded by a consonant and followed by a vowel.Frames associated with the same class may be expected to have similaracoustic properties. It is known that speech manipulation techniquesusing class dependent transformations of frames may perform better thanthe ones employing global class independent transformations.

Evaluation of Statistical Evidence and Modeling of the Pitch-IntensityRelationship

The pitch-intensity relationship may be analyzed per speech class usinga TTS voice dataset built by techniques known in the art. The analysispresented below was performed using a TTS voice dataset built fromexpressive sports news sentences uttered by a female American Englishspeaker and recorded at a 22050 Hz sampling rate.

As a starting phase of the voice dataset building procedure, the speechsignals were analyzed at the frame update rate of 5 ms (i.e., framesize=5 ms). The analysis included pitch contour estimation using analgorithm similar to the one disclosed in A. Sorin et al., “The ETSIExtended Distributed Speech Recognition (DSR) standards: client sideprocessing and tonal language recognition evaluation”, In Proc. ICASSP2004, Montreal, Quebec, Canada, May 2004. The analysis also includedestimation of pitch harmonic magnitudes, also known as line spectrum,using the method presented in D. Chazan et al, “High quality sinusoidalmodeling of wideband speech for the purpose of speech synthesis andmodification”, In Proc. ICASSP 2006, Toulouse, France, May 2006. Theline spectrum estimation for each frame was performed using the 2.5pitch period long hamming windowing function centered at the framecenter. Then Mel-frequency Regularized Cepstral Coefficients (MRCC)spectral envelope parameters vector (see S. Shechtman and A. Sorin,“Sinusoidal model parameterization for Hidden Markov Model (HMM)-basedTTS system”, in Proc. Interspeech 2010, Makuhari, Japan, September 2010)was calculated for each frame. It should be noted that the specificpitch and line spectrum estimators and the MRCC spectral parameters maybe substituted by other estimators known in the art and spectralparameters, respectively.

The frames represented by the MRCC vectors were used in a standardHMM-based phonetic alignment and segmentation procedure with three HMMstates per phoneme. Hence, each segment may represent one third of aphoneme. The speech classes generation and segment classification wereperformed using a standard binary decision tree approach depending onphonetic-linguistic context and the MRCC vectors homogeneity. Thisprocess yielded about 5000 speech classes.

Only fully-voiced segments, i.e. the ones comprised of all voiced frames(i.e., frames which represent periodic parts of the utterance), wereused for the pitch-intensity relationship analysis. The classescontaining less than 5 frames found in fully voiced segments wereexcluded from the analysis. This pruning procedure retained about 1900classes containing in total more than 1.6 million frames which sums tomore than 8000 seconds of purely voiced speech material.

The energy was estimated for each frame as:

$\begin{matrix}{E = {\sum\limits_{i = 1}^{N_{h}}\; A_{i}^{2}}} & (1)\end{matrix}$

where A_(i) is the magnitude of the i-th harmonic and N_(h) is thenumber of harmonics in the full frequency band (up to the Nyquistfrequency of 11025 Hz) for that frame excluding the direct currentcomponent, i.e. excluding the harmonic associated with zero frequency.It should be noted that the line spectrum estimation algorithm (see D.Chazan et al, id) yields the harmonic magnitudes scaled in such a waythat:

$\begin{matrix}{E \approx {\frac{1}{2\; T}{\sum\limits_{n = 1}^{T}\; {{\overset{\sim}{s}}^{2}(n)}}}} & (2)\end{matrix}$

where {tilde over (s)}(n), n=1, 2, . . . , T is a representative pitchcycle associated with the frame center, and T is the rounded pitchperiod in samples corresponding to the F0 value associated with theframe. Thus the energy given by (1) is proportional to the averageper-sample energy calculated over the representative pitch cycle derivedfrom the speech signal around the frame center. Estimation of therepresentative pitch cycle is addressed herein below. Another linespectrum estimator may be used, e.g., spectral peak picking. In such acase, the appropriate scaling factor may be introduced in (1) toreproduce the results of the current analysis.

The intensity I was calculated for each frame as the squared root of theenergy:

I=√{square root over (E)}  (3)

Thus a frame k may be represented by the intensity I_(k) and the pitchfrequency F0 _(k). Both the parameters may be mapped to a logarithmicscale aligned with the human perception of the loudness and tone changesmeasured in decibel (dB) and octave (oct) units respectively:

IdB_(k)=20·log₁₀ I _(k)  (4)

F0oct_(k)=log₂ F0_(k)  (5)

Reference is now made to FIG. 1, which shows a linear regression graphpresenting the correlation between pitch frequency and intensity withina speech class derived from the TTS voice dataset. The frames associatedwith one of the speech classes are depicted as points on the (F0 oct,IdB) plane. The distribution of the points in FIG. 1 illustrates asalient interdependency between the intensity and pitch frequency.

To quantify the degree of this interdependency, the correlationcoefficients between {IdB_(k)} and {F0 oct_(k)} sequences werecalculated at class levels and over the voice dataset globally. Forexample, the correlation coefficient calculated within the class shownin FIG. 1 is 0.67. The averaging of the intra-class correlationcoefficients weighted by the respective class frame counts yielded thevalue of 0.52. The correlation coefficient calculated over the whole setof more than 1.6 million frames, ignoring the class association, is0.40.

The correlation measures obtained in the above evaluation may provide astatistical evidence of the pitch-intensity relationship. An analyticalexpression which models the pitch-intensity relationship utilizinglog-linear regression is herein disclosed. Nevertheless, othermathematical representations may be utilized, such as piece-wise linear,piece-wise log-linear or exponential functions. With reference to FIG.1, the solid line is the regression line. The pitch-intensityrelationship may be then expressed within each class C as:

IdB_(C)=λ_(1C) ·F0oct_(C)+λ_(2C)  (6)

The least squares approximation yields the following values for theregression parameters λ_(1C) and λ_(2C):

$\begin{matrix}{\mspace{79mu} {\lambda_{1\; C} = \frac{{\sum\limits_{\forall{k \in C}}\; {F\; 0\; {{oct}_{k} \cdot {\sum\limits_{\forall{k \in C}}\; {IdB}_{k}}}}} - {N_{C}{\sum\limits_{\forall{k \in C}}\; {F\; 0\; {{oct}_{k} \cdot {IdB}_{k}}}}}}{\lbrack {\sum\limits_{\forall{k \in C}}\; {F\; 0\; {oct}_{k}}} \rbrack^{2} - {N_{C} \cdot {\sum\limits_{\forall{k \in C}}\; {F\; 0\; {oct}_{k}^{2}}}}}}} & (7) \\{\lambda_{2\; C} = \frac{{\sum\limits_{\forall{k \in C}}\; {F\; 0\; {{oct}_{k} \cdot {\sum\limits_{\forall{k \in C}}\; {F\; 0\; {{oct}_{k} \cdot {IdB}_{k}}}}}}} - {\sum\limits_{\forall{k \in C}}\; {F\; 0\; {{oct}_{k}^{2} \cdot {\sum\limits_{\forall{k \in C}}\; {IdB}_{k}}}}}}{\lbrack {\sum\limits_{\forall{k \in C}}\; {F\; 0\; {oct}_{k}}} \rbrack^{2} - {N_{C} \cdot {\sum\limits_{\forall{k \in C}}\; {F\; 0\; {oct}_{k}^{2}}}}}} & (8)\end{matrix}$

where N_(C) is the number of frames associated with the class C.

The model given by equations (6), (7) and (8) may be used for predictionof the intensity of a frame from the pitch frequency associated with theframe. Additionally this model may be used for prediction of the frameintensity altering when the frame pitch frequency is modified by a givenamount.

The slope coefficient λ_(1C) given by (7) of the regression line (6) mayindicate the average constant rate of intensity increase dependent onpitch increase. For example, for the class shown in FIG. 1, the slopecoefficient is equal to 9.7 dB/octave. This means that for speech framesassociated with the concerned class, the intensity increases by 9.7 dBwhen the fundamental frequency rises by the factor of two. As wasexpected, the slope value may vary between the classes. Averaging of theintra-class slope values weighted by the respective class counts yieldedthe average slope value of 4.5 dB/octave. The average slope calculatedover the whole set of more than 1.6 million frames, ignoring the classassociation, is 6.0 dB/octave.

The same evaluation performed with another TTS voice dataset derivedfrom neutral speech produced by another American English speaker yieldedsimilar results: the weighted average of the intra-class correlationcoefficients was 0.40 and the weighted average of the intra-class slopevalues was 4.6 dB/octave.

Coherent Modification of Pitch Contour and Intensity Contour

Reference is now made to FIGS. 2A and 2B. FIG. 2A shows a flowchart of amethod, constructed and operative in accordance with an embodiment ofthe disclosed technique. FIG. 2B shows a flowchart of an exemplarymethod, constructed and operative in accordance with another embodimentof the disclosed technique. FIG. 2B shows a method which is exemplary tothe general method of FIG. 2A. The method of FIG. 2B describes pitchcontour and intensity contour modification based on frame classificationinformation. In order to simplify the description, the general methodsteps are described herein for both methods, while indicating thedifferences between the two. Those of skill in the art will recognizethat one or more of the steps of the methods of FIGS. 2A and/or 2B maybe omitted, while still providing advantageous, coherent pitch andintensity modification. In addition, the method steps disclosed hereinmay be carried in an order different than as listed below.

In steps 100 and 200, an utterance, indicated U_(org) (i.e., theoriginal utterance), may be received (also referred as the originalutterance, the received utterance or the received original utterance).The utterance may include a plurality of consecutive segments. Eachsegment may include one or more consecutive frames. Thus, the utterancemay include a plurality of consecutive frames, while at least one of theframes is a voiced frame.

The utterance may be produced, for example, by a human speaker (i.e.,natural speech) or by a TTS system (i.e., synthesized speech). Theutterance may be represented by a raw speech signal {s_(org)(n), n=0, 1,. . . N} or by a sequence {P_(k), k=0, 1, . . . , K} of frame-wise setsof vocoder parameters as described hereinabove.

Modern TTS systems may employ either concatenative (also known as unitselection) synthesis scheme or a statistical synthesis scheme or a mixof them. Regardless of the TTS scheme, the synthesized utterance may becomposed of segments. A segment may include one or more consecutivespeech frames. In the concatenative TTS, the segments (representedeither by raw data or parameterized) may be extracted from naturalspeech signals. In statistical TTS, the segment frames may berepresented in a parametric form and the vocoder parameter sets may begenerated from statistical models. A TTS system may include a voicedataset, which may include a repository of labeled speech segments or aset of statistical models or both.

The hierarchical dichotomies described above are summarized in Table 1below.

TABLE 1 Application context of pitch modification TTS ConcatenativeStatistical Natural speech Parametric X X X representation Raw speechsignal X Not applicable X

In some embodiments, each frame of the received original utterance maybe mapped to a distinct speech class selected from a predefined set ofspeech classes. Thus, according to step 200, a sequence: {C_(k), k=0, 1,. . . , K} of the frame class identity labels may be received along withthe original utterance.

In the context of TTS, the frame classification information may beinherently available. A TTS voice dataset may include a collection ofspeech segments (raw, parameterized or represented by statisticalmodels) clustered according to their phonetic-linguistic context. Atsynthesis time, the received utterance may be composed of segmentsselected based on their class labels. The frames included in a segmentmay be mapped to the class associated with the segment.

In embodiments relating to applications operating on natural speech, theframe classification information may be further generated according tothe disclosed method. Each of the frames of the utterance may be mappedto a corresponding speech class selected from a predefined set of speechclasses. Alternatively, such information may be received with theoriginal utterance. For example, an Automatic Speech Recognition (ASR)process may be applied to the natural utterance. The ASR may provide aword transcript and phonetic identity of each speech frame. NaturalLanguage Processing (NLP) techniques may be applied to the wordtranscript to extract required linguistic features such as part ofspeech, part of sentence, etc. Finally, the phonetic-linguistic featuresassociated with each frame may be mapped to a certain predefinedphonetic-linguistic context as it is done in TTS front-end blocks.

As part of steps 100 and/or 200, the original pitch contour and/or thetarget pitch contour of the received utterance may be also received. Theoriginal pitch contour (i.e., prior to modification of the receivedutterance) may include a sequence of original pitch frequency values,corresponding to the frames of the received utterance, and may beindicated as: {F0 _(org)(n_(k)), k=0, 1, 2, . . . , K}. The target pitchcontour may include a sequence of target pitch frequency values,corresponding to the frames of the received utterance, and may beindicated as: {F0 _(out)(n_(k)), k=0, 1, 2, . . . , K}. Optionally, theoriginal pitch contour and/or the target pitch contour may be measuredor generated as part of the disclosed technique. The original pitchcontour may be measured and the target pitch contour may be generated invarious manners depending on whether the utterance was produced by ahuman speaker, a concatenative TTS system or by a statistical TTSsystem.

When natural speech is concerned, the original pitch contour may bemeasured from the speech signal corresponding to the original utteranceby one of the pitch estimation algorithms known in the art, such as theone disclosed in A. Sorin et al. (2004), id. When concatenative TTS isconcerned, a concatenation of the segmental pitch contours may beperformed. When statistical TTS is concerned, the original pitch contourmay be the trajectory of the F0 parameter. The trajectory may begenerated from the statistical models trained on the speech dataset,which is used for the modeling of all the vocoder parameters in thesystem.

When natural speech and TTS are concerned, the target pitch contour maybe generated by transforming the original pitch contour. For example: F0_(out)(n_(k))=avF0 _(out)+α·[F0 _(org)(n_(k))−avF0 _(org)], where avF0_(org) and avF0 _(out) are the utterance level average values of theoriginal and target pitch contours respectively and a is the parameterthat controls the dynamics and hence influences the perceivedexpressiveness of the target pitch contour. The utterance level averageavF0 _(out) may be set to a desired pre-defined value or made dependenton the average value of the original pitch contour, e.g. avF0_(out)=β·avF0 _(org) where β is a control parameter. When TTS(concatenative and statistical) is concerned, the target pitch contourmay be generated by a rule-based framework. Alternatively, it may bederived from a relevant, typically expressive and relatively smallspeech data corpus external to the voice dataset used for the synthesis.The desired pitch values may be generated initially at a lower temporalresolution than a frame, e.g. one value per phoneme, and thendownsampled to the frame centers grid.

Alternatively, the utterance in its original form and the utterance witha modified pitch (i.e. pitch-modified utterance) may be received. Theoriginal pitch contour may be then measured based on the utterance inits original form and/or the target pitch contour may be measured basedon the pitch-modified utterance.

In steps 110 and 210, an original intensity contour {I_(org)(n_(k)),k=0, 1, 2, . . . , K} of the utterance may be calculated. Thiscalculation may be performed by applying an instantaneous intensityestimator to the original utterance.

The instantaneous intensity estimator may be defined to output a valueI(n) proportional to the average amplitude of samples of a speech signalcorresponding to an utterance within a short time window surrounding thediscrete time instant n.

When the representation of the utterance is based on a Sinusoidal Model,an instantaneous intensity estimator operating in frequency domain maybe defined as specified by equations (3) and (1). The harmonicmagnitudes A_(i) associated with the frame centered at the time instantn may be determined using the line spectrum estimation algorithm similarto the D. Chazan et al, id. If the line spectrum is determined by aShort Time Fourier Transform (STFT) peak picking algorithm, then theharmonic magnitudes may be divided by the DC (direct current) value ofthe spectrum of the windowing function used in the STFT.

In some embodiments, the intensity estimator may be defined to operateover a speech signal in time domain. In this case the intensity I(n) maybe also calculated as the square root of the energy E(n) as specified inequation (3), but the energy may be estimated from the speech signal susing a time window surrounding the time instant n. The simplest form ofthe energy estimation may be given by:

$\begin{matrix}{{E(n)} = {\frac{1}{L_{n}}{\sum\limits_{i = {n - {\lfloor{{({L_{n} - 1})}/2}\rfloor}}}^{i = {n - {\lfloor{{({L_{n} - 1})}/2}\rfloor} + L_{n} - 1}}\; {s^{2}(i)}}}} & (9)\end{matrix}$

where L_(n) is the window length (generally frame dependent) and └.┘denotes the down integer rounding operation. The value of L_(n) may beset so that the window includes one pitch cycle:

L _(n) =T _(n) =[F _(s) /F0(n)]  (10)

where F_(s) is the sampling frequency, e.g. 22,050 Hz, and [.] denotesthe integer rounding operation. However, other settings may be possible.With these settings the expression (9) may represent the average energyof the pitch cycle centered at the time instant n.

A more robust method may be employed to extract a representative pitchcycle {tilde over (s)}(i), i=1, 2, . . . , T_(n) as a weighted averageof the pitch cycles occurring in proximity of the time instant n:

$\begin{matrix}{{\overset{\sim}{s}(i)} = \frac{\sum\limits_{k \in {K{(i)}}}\; {{s( {n - \lbrack {T_{n}/2} \rbrack + i + {kT}_{n}} )} \cdot {w( {n - \lbrack {T_{n}/2} \rbrack + i + {kT}_{n}} )}}}{\sum\limits_{k \in {K{(i)}}}\; {w( {n - \lbrack {T_{n}/2} \rbrack + i + {kT}_{n}} )}}} & ( {11a} )\end{matrix}$

where:w(i), i=1, 2, . . . , 2M+1 is a positive windowing function symmetricrelatively to i=M+1, e.g., Hamming windowing function; andthe interval K(i) spanned by the summation index k is defined so that:|−[T_(n)/2]+i+kT_(n)|≦M.Then the energy E(n) may be calculated as:

$\begin{matrix}{{E(n)} = {\frac{1}{T_{n}}{\sum\limits_{i = 1}^{T_{n}}\; {{\overset{\sim}{s}}^{2}(i)}}}} & ( {11b} )\end{matrix}$

In steps 120 and 220, a pitch-modified utterance U_(intr) (hereafterreferred to as intermediate utterance) may be generated based on thetarget pitch contour. The pitch-modified utterance may be generated byapplying a pitch modification technique to the original utteranceU_(org). Any pitch modification technique suitable for the originalutterance representation form may be applied. Some pitch modificationtechniques depending on the underlying speech representation areexemplified below.

When raw speech signal is concerned, the pitch modification may beperformed by using Pitch Synchronous Overlap and Add (PSOLA) techniqueseither in time or frequency domain (see E. Moulines, W. Verhelst,“Time-domain and frequency-domain techniques for prosodic modificationof speech”, in Speech Coding and Synthesis, B. Klein ed, ElsevierScience Publishers 1995). PSOLA may require calculation of pitch marks(i.e., pitch cycle related epochs) derived from the pitch contour andthe signal.

When a parameterized signal is concerned, the pitch modification may beperformed in the space of vocoder parameters yielding a parameterizedpitch-modified signal. In some embodiments it may be preferable tofurther convert the parameterized pitch-modified (or intermediate)signal to the form of raw speech signal. The pitch modificationalgorithm may depend on the parametric representation adopted in thesystem.

Some of the statistical TTS systems may adopt the source-filterrepresentation framework (see Zen, H., Tokuda, K., and Black, A. W.,“Statistical parametric speech synthesis”, Speech Communication, vol.51, November 2009, pp. 1039-1064) where the source may represent theexcitation signal produced by the vocal folds and the filter mayrepresent the vocal tract. Within this framework the main(quasi-periodic) part of the excitation signal may be generated usingthe target pitch contour. Other vocoding techniques may employframe-wise sinusoidal representations (see Chazan et al., id; S.Shechtman and A. Sorin, “Sinusoidal model parameterization for HMM-basedTTS system”, in Proc. Interspeech 2010, Makuhari, Japan, September 2010;T. F. Quatieri and R. I. McAulay, “Speech Transformations Based on aSinusoidal Representation.” IEEE Trans. Acoust. Speech Signal Process.ASSP-34, 1449, Dec. 1986; Stylianou, Y., et al, “An extension of theadaptive Quasi-Harmonic Model”, in Proc. ICASSP 2012, Kyoto Japan, March2012). In such a framework, the pitch modification at frame k may beperformed by an interpolation and re-sampling of the frame-wise harmonicstructures (also known as line spectra) along the frequency axis at theintegral multiples of the target pitch frequency F0 _(out)(k).

The intermediate utterance U_(intr), may be represented either in aparametric form as a sequence {P_(intr)(n_(k)), k=0, 1, . . . , K} ofmodified sets of vocoder parameters or by a modified raw speech signals_(intr)(n), depending on the representation type of the receivedoriginal utterance. Although other representations may be applicable, inembodiments based on Sinusoidal Model representation of the originalutterance, the intermediate utterance may be kept in the parameterizedform while otherwise, it may be represented by a raw speech signal.

In steps 130 and 230, an intensity modification factor ρ(n_(k)) may becalculated for each frame based on the original pitch contour and thetarget pitch contour. More specifically, an intensity modificationfactor ρ(n_(k)) may be calculated for each frame k based on the originalpitch frequency value F0 _(org)(n_(k)) and the target pitch frequencyvalue F0 _(out)(n_(k)) associated with that frame. If the frame isunvoiced then the intensity modification factor may be set such that:ρ(n_(k))=1.

For a voiced frame the calculation of the intensity modification factormay be based on a pitch-to-intensity transformation R modeling therelationship between the instantaneous pitch frequency F0 andinstantaneous intensity I. Such a transformation may be defined as:

I=R(F0,λ)  (12)

where λ is a set of control parameters. Thus, the transformation may berepresented, for example, as a function of the pitch frequency and a setof control parameters.

The calculation of the intensity modification factor ρ(n_(k)) for avoiced frame k may include setting values for the control parameters λof the pitch-to-intensity transformation R.

According to step 230, the calculation of the intensity modificationfactor may be further based on the frame classification information. Thecontrol parameters may be then set to a value λ_(C) _(k) * associatedwith the speech class C_(k) corresponding to the frame. The value λ_(C)_(k) * for each speech class may be fetched from an Intensity PredictionModels Database. Such a database may be generated in an offline modelingstep 260, as described herein below. If the class C_(k) is marked at theoffline modeling step as irrelevant for the intensity modification thenthe intensity modification factor may be directly set as ρ(n_(k))=1without setting the control parameter values λ.

With reference to both methods, i.e., the methods of FIGS. 2A and 2B,the control parameters may be set to a constant predefined value. Thepredefined value may be selected by using a subjective listeningevaluation. Alternatively, the predefined value may be determined as theaverage value derived from an existing clustered collection of speechusing an evaluation similar to the one presented in the section entitled“Evaluation of Statistical Evidence and Modeling of the Pitch-IntensityRelationship” hereinabove.

The calculation of the intensity modification factor may further includecalculating a reference value of the intensity I_(org) ^(ref)(n_(k))corresponding to the original pitch frequency applying theR-transformation of equation (12) to the original pitch frequency asfollows:

I _(org) ^(ref)(n _(k))=R(F0_(org)(n _(k)),λ)  (13)

The calculation of the intensity modification factor may further includecalculating a reference value of the intensity I_(mod) ^(ref)(n_(k))corresponding to the modified (i.e., target) pitch frequency applyingthe R-transformation of equation (12) to the modified pitch frequency asfollows:

I _(mod) ^(ref)(n _(k))=R(F0_(out)(n _(k)),λ)  (14)

The intensity modification factor ρ(n_(k)) may be then obtained bydividing the reference value of the intensity I_(mod) ^(ref)(n_(k))corresponding to the target (i.e., modified) pitch frequency by thereference value of the intensity I_(org) ^(ref)(n_(k)) corresponding tothe original pitch frequency:

ρ(n _(k))=I _(mod) ^(ref)(n _(k))/I _(org) ^(ref)(n _(k))  (15)

In some embodiments, the R-transformation (12) may be defined accordingto equations (4), (5) and (6), i.e., based on log-linear regression, asfollows:

I=10^(1/20(λ) ¹ ^(log) ² ^(F0+λ) ² ⁾  (16)

yielding the intensity modification factor in the form:

ρ(n _(k))=10^(λ) ¹ ^([log) ² ^(F0) ^(out) ^((n) ^(k) ^()−log) ² ^(F0)^(org) ^((n) ^(k) ^()])  (17)

Hence, in such embodiments, the intensity modification factor dependsonly on the amount of pitch modification F0 _(out)/F0 _(org) regardlessof the absolute level of F0. With specific reference to the method ofFIG. 2B, the value of the parameter λ₁ may be set to the valuecalculated in step 260 for the corresponding speech class. With specificreference to the method of FIG. 2A, the parameter may be set to theaverage λ₁ value of 6 dB/octave derived from an existing speech corpus.Thus, the intensity modification factor may be equal to ten in the powerof the twentieth of an empiric average decibels per octave ratio of aspeech signal (i.e., λ₁) multiplied by the extent of pitch modificationexpressed in octaves.

In some embodiments, other types of the R-transformation may be used,e.g. based on an exponential function, or piece-wise linear function inthe linear or log scales.

In an optional step, temporal smoothing may be applied to the sequence{ρ(n_(k)), k=0, 1, 2, . . . , K} of the intensity modification factors.A smoothened sequence {ρ_(s)(n_(k)), k=0, 1, 2, . . . , K} of theintensity modification factors may be then generated. Such temporalsmoothing may prevent abrupt changes in the final intensity contour.

Any smoothing technique, as known in the art, may be adopted. One choicemay be the weighted moving averaging method, i.e. the convolution with asymmetric positive (2I+1)-tap filter v:

$\begin{matrix}{{{{\rho_{s}( n_{k} )} = \frac{\sum\limits_{i = {- I}}^{i = I}\; {v_{i} \cdot {\rho ( n_{k + i} )}}}{\sum\limits_{i = {- I}}^{i = I}\; v_{i}}};{v_{i} > 0}},{v_{i} = v_{- i}}} & (18)\end{matrix}$

The filter v may be defined, for example, as: [v₀=3, v₁=2, v₂=1]. Thesmoothed sequence of the intensity modification factors may be used inthe following steps instead of the original sequence of the intensitymodification factors.

In steps 140 and 240, a final intensity contour {I_(out)(n_(k)), k=0, 1,2, . . . , K} may be calculated. The calculation may be based on theoriginal intensity contour and the sequence of the intensitymodification factors.

The sequence of the intensity modification factor values (i.e., perframe) may be applied to the original intensity contour:

I _(out)(n _(k))=I _(org)(n _(k))·ρ(n _(k))  (19)

The values of the final intensity contour may be then limited in orderto preserve the original amplitude range of the output speech signaland/or to prevent possible clipping of the output signal:

I _(out)(n _(k))=max(I _(out)(n _(k)),I _(max))  (20)

where I_(max) is either a predefined value or is derived from theoriginal utterance, for example:

$\begin{matrix}{I_{\max} = {\max\limits_{0 \leq k \leq K}{I_{org}( n_{k} )}}} & (21)\end{matrix}$

In steps 150 and 250, a coherently-modified speech signal (i.e., theoutput speech signal) may be generated by time-dependent intensityscaling of the intermediate (i.e. pitch-modified) utterance according tothe final intensity contour.

The intensity scaling may include determining an intermediate intensitycontour {I_(intr)(n_(k)), k=0, 1, 2, . . . , K} based on thepitch-modified utterance. The intermediate intensity contour may bedetermined by applying the instantaneous intensity estimator, describedhereinabove, to the intermediate (i.e. pitch modified) utterance.

The intensity scaling may further include determining a gain factorcontour {g(n_(k)), k=0, 1, 2, . . . , K} comprised of frame-wise gainfactor values:

$\begin{matrix}{{g( n_{k} )} = \frac{I_{out}( n_{k} )}{I_{intr}( n_{k} )}} & (22)\end{matrix}$

In embodiments employing Sinusoidal Model based speech representation,the intensity scaling may further include multiplying all the sinusoidalcomponent magnitudes by g(n_(k)) for each frame k of the pitch-modifiedutterance and then transforming of the modified parametricrepresentation to the output speech signal s_(out)(n).

Otherwise, the intensity scaling may further include deriving a gainfactor signal {g(n), n=0, 1, 2, . . . , N} by down-sampling the gainfactor contour {g(n_(k)), k=0, 1, 2, . . . , K} using an interpolationfunction h:

g(n)=h(n−n _(k))·g(n _(k))+(1−h(n−n _(k)))·g(n _(k+1)) for n _(k) ≦n≦n_(k+1)1=h(0)>h(1)>h(2) . . . >h(n _(k+1) −n _(k))=0  (23)

The interpolation function {h(i), i=0, 1, . . . , t_(k+1)−t_(k)} may beset, for example, to the right half of hann windowing function.

The intensity scaling may further include multiplying the raw speechsignal corresponding to the intermediate utterance (i.e. thepitch-modified speech signal or the intermediate speech signal) by thegain factor signal:

s _(out)(n)=s _(intr)(n)·g(n)  (24)

With specific reference to FIG. 2B and in an optional step 260, anoffline modeling of the pitch-intensity relationship may be performed toreceive values for the control parameters according to speech classes.This step may follow the description provided in the section entitled“Evaluation of Statistical Evidence and Modeling of the Pitch-IntensityRelationship” hereinabove.

A collection of speech data may be required for this step. Thecollection of speech data may include speech segments mapped to thespeech classes expected in the frame classification information. Eachspeech segment may be divided to frames. A frame may be mapped to thespeech class associated with the segment including that frame. In thecontext of a TTS application, such a collection may be readily availableas a part of the voice dataset. In the context of natural utteranceprocessing, such a collection may be composed from a transcribed andphonetically aligned single speaker data corpus either publicallyavailable (e.g., TIMIT Acoustic-Phonetic Continuous Speech Corpus,Linguistic Data Consortium. University of Pennsylvania. Web. Mar. 1,2015, https://catalog.ldc.upenn.edu/LDC93S1) or proprietary.

For each class C, all the voiced frames may be gathered. In order tomake the estimation more robust, the voiced frames which are included innot fully-voiced segments (i.e., a segment containing at least oneunvoiced frame) may be excluded. The following sub-steps may beperformed for a class C, optionally provided that the number of framesgathered for this class is greater than a predefined amount, e.g., four.

Each analyzed frame k may be represented by the pitch frequency F0 _(k)value and the intensity I_(k) value estimated by using the intensityestimator adopted for the specific case. Two observation vectorsI_(C)={I_(k), ∀kεC} and F0 _(C)={F0 _(k), ∀kεC} may be generated bystacking together the frame intensity and pitch frequency valuesrespectively. The optimal set of the parameter values λ_(C)* of thepitch-to-intensity transformation (12) may be determined such that theintensity values predicted by the equation (12) yields the best possibleapproximation of the real intensity values observed in the framesassociated with class C:

$\begin{matrix}{\lambda_{C}^{*} = {\underset{\lambda}{argmin}{{{W_{I}( I_{C} )} - {W_{F\; 0}( {R( {{F\; 0_{C}},\lambda} )} )}}}^{2}}} & (25)\end{matrix}$

where W_(I)(x) and W_(F0)(x) are intensity and pitch frequency scalemapping functions respectively. It should be noted that W_(I)(X) andW_(F0)(X) denote the component-wise transformations of the vector X;R(X,λ) denote the component-wise transformation of the vector X. In someembodiments, where no scale mapping is performed, an identity scalemapping function may be used (e.g., W_(I)(x)=x). The optimizationproblem (25) may be solved by a suitable numeric optimization technique,as known in the art.

In some embodiments, the R-transformation may be defined by equation(16) and the scale mapping functions may be defined according toequations (4) and (5), as follows:

W _(I)(I)=IdB=20·log₁₀ I  (26a)

W _(F0)(F0)=F0oct=log₂ F0  (26b)

In this case, the optimization problem of equation (25) may be solvedanalytically and the optimal parameter values for a class C may becalculated as specified by equations (7) and (8).

The per-class optimal control parameter values κ_(C)* labeled by therespective class identity may be stored in the intensity predictionmodels database which may be available at run-time.

Speech classes that do not include enough observation data for thestatistically meaningful estimation of the parameters 2 (for example,those classes containing less than five frames found in fully voicedsegments) may be marked in the intensity prediction models database asirrelevant for the intensity modification.

Step 260 may be performed offline and prior to the performance of steps200-250 of the method of FIG. 2B.

Reference is now made to FIG. 3, which shows an exemplary system 300,according to an embodiment. System 300 may include a computing device310. Computing device 310 may include a hardware processor 320 and astorage device 330. System 300 may further include an input/output(“I/O”) device 340 and/or an intensity prediction models database 350(or simply ‘database 350’). Hardware processor 320 may include one ormore hardware processors, storage device 330 may include one or morestorage devices, 110 device 340 may include one or more 110 devices anddatabase 350 may include one or more databases. Hardware processor 320may be configured to execute the methods of FIGS. 2A and/or 2B and, tothis end, be in communication with database 350 and receive datatherefrom. Database 350 may include values of control parameters per aspeech class which may be used for generating a coherently modifiedspeech signal according to the method of FIG. 2B. I/O device 340 may beconfigured to allow a user to interact with system 300. For example, I/Odevice 340 may include, inter alia, a microphone to allow a user torecord and thus provide an utterance to computing device 310. Theutterance may be then stored in storage device 330 and processed byhardware processor 320 according to the methods of FIGS. 2A and/or 2B.Dedicated software, implementing the methods of FIGS. 2A and/or 2Bdiscussed above, may be stored on storage device 330 and executed byhardware processor 320.

Database 350 may be stored on any one or more storage devices such as aFlash disk, a Random Access Memory (RAM), a memory chip, an opticalstorage device such as a CD, a DVD, or a laser disk; a magnetic storagedevice such as a tape, a hard disk, storage area network (SAN), anetwork attached storage (NAS), or others; a semiconductor storagedevice such as Flash device, memory stick, or the like. Database 350 maybe a relational database, a hierarchical database, object-orienteddatabase, document-oriented database, or any other database.

In some embodiments, computing device 310 may include an I/O device 340such as a terminal, a display, a keyboard, a mouse, a touch screen, amicrophone, an input device and/or the like, to interact with system300, to invoke system 300 and to receive results. It will however beappreciated that system 300 may operate without human operation andwithout I/O device 340.

In some exemplary embodiments of the disclosed subject matter, storagedevice 330 may include or be loaded with code for a user interface. Theuser interface may be utilized to receive input or provide output to andfrom system 300, for example receiving specific user commands orparameters related to system 300, providing output, or the like.

Experimental Results

A standard subjective listening preference test was performed in orderto test the disclosed speech modification. Twelve text messagesconveying expressive contents (Sports News) were synthesized by a TTSsystem trained on a neutral voice. During the synthesis, expressivepitch contours where implanted into the speech signals, i.e. the defaultpitch contours emerging from the TTS system were replaced by externallygenerated expressive ones. The synthesis has been performed twice: A)while preserving the original energy contour; and B) while modifying theenergy contour in accordance with the disclosed technique. Ten listenerswere presented with 12 pairs of stimuli each. Each pair included theabove version A and version B of the same stimulus. After listening toboth versions of a stimulus, a listener was instructed to choose betweenfive options: no preference, preference to either version, strongpreference to either version. The evaluation revealed average preferenceof 51% including 14% of strong preference for the version B, i.e., thespeech signal modified according to the disclosed technique, and only25% for the version A.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A method comprising: receiving an utterance, anoriginal pitch contour of the utterance, and a target pitch contour forthe utterance, wherein the utterance comprises a plurality ofconsecutive frames, and wherein at least one of said frames is a voicedframe; calculating an original intensity contour of said utterance;generating a pitch-modified utterance based on the target pitch contour;calculating an intensity modification factor for each of said frames,based on said original pitch contour and on said target pitch contour,to produce a sequence of intensity modification factors corresponding tosaid plurality of consecutive frames; calculating a final intensitycontour for said utterance by applying said intensity modificationfactors to said original intensity contour; and generating acoherently-modified speech signal by time-dependent scaling of theintensity of said pitch-modified utterance according to said finalintensity contour.
 2. The method of claim 1, wherein the receivedutterance is natural speech, and wherein the method further comprisesmapping each of said frames to a corresponding speech class selectedfrom a predefined set of speech classes.
 3. The method of claim 1,wherein the calculating of the intensity modification factor for each ofsaid frames is based on a pitch-to-intensity transformation modeling therelationship between the instantaneous pitch frequency and theinstantaneous intensity of the utterance, and wherein thepitch-to-intensity transformation is represented as a function of apitch frequency and a set of control parameters.
 4. The method of claim3, wherein each of said frames is mapped to a corresponding speech classselected from a predefined set of speech classes, and wherein the methodfurther comprising setting the values of said control parameters foreach of said frames according to its corresponding speech class.
 5. Themethod of claim 4, further comprising offline modeling of the pitch tointensity relationship to receive said values for said controlparameters according to said speech classes.
 6. The method of claim 3,further comprising setting said control parameters to constantpredefined values.
 7. The method of claim 3, wherein thepitch-to-intensity transformation is based on log-linear regression, andwherein the set of control parameters comprises the slope coefficient ofthe regression line of the log-linear regression.
 8. The method of claim3, wherein the intensity modification factor is ten in the power of thetwentieth of the ratio of average empirical decibels per octavemultiplied by the extent of pitch modification expressed in octaves. 9.The method of claim 8, wherein the value of the ratio of empiricaldecibels per octave is set to six decibels per octave.
 10. The method ofclaim 3, wherein the calculating of the intensity modification factorfor each of said frames comprises: calculating a reference value of theintensity corresponding to an original pitch frequency of the originalpitch contour for said each frame, by applying the pitch-to-intensitytransformation to the original pitch frequency; calculating a referencevalue of the intensity corresponding to the target pitch frequency ofthe target pitch contour for each of said frames by applying thepitch-to-intensity transformation to the target pitch frequency; anddividing the reference value of the intensity corresponding to thetarget pitch frequency by the reference value of the intensitycorresponding to the original pitch frequency.
 11. A computer programproduct comprising a non-transitory computer-readable storage mediumhaving program code embodied therewith, the program code executable byat least one hardware processor to: receive an utterance, an originalpitch contour of the utterance, and a target pitch contour for theutterance, wherein the utterance comprises a plurality of consecutiveframes, and wherein at least one of said frames is a voiced frame;calculate an original intensity contour of said utterance; generate apitch-modified utterance based on the target pitch contour; calculate anintensity modification factor for each of said frames, based on saidoriginal pitch contour and on said target pitch contour, to produce asequence of intensity modification factors corresponding to saidplurality of said consecutive frames; calculate a final intensitycontour for said utterance by applying said intensity modificationfactors to said original intensity contour; and generate acoherently-modified speech signal by time-dependent scaling of theintensity of said pitch-modified utterance according to said finalintensity contour.
 12. The computer program product of claim 11, whereinthe received utterance is natural speech, and wherein the program codeis further executable by said at least one hardware processor to mapeach of said frames to a corresponding speech class selected from apredefined set of speech classes.
 13. The computer program product ofclaim 11, wherein the calculating of the intensity modification factorfor each of said frames is based on a pitch-to-intensity transformationmodeling the relationship between the instantaneous pitch frequency andthe instantaneous intensity of the utterance, and wherein thepitch-to-intensity transformation is represented as a function of apitch frequency and a set of control parameters.
 14. The computerprogram product of claim 13, wherein each of said frames is mapped to acorresponding speech class selected from a predefined set of speechclasses, and wherein the program code is further executable by said atleast one hardware processor to set the values of said controlparameters for each of said frames according to its corresponding speechclass.
 15. The computer program product of claim 14, wherein the programcode is further executable by said at least one hardware processor tooffline model the pitch to intensity relationship to receive said valuesfor said control parameters according to said speech classes.
 16. Thecomputer program product of claim 13, wherein the pitch to intensitytransformation is based on log-linear regression, and wherein the set ofcontrol parameters comprises the slope coefficient of the regressionline of the log linear regression.
 17. The computer program product ofclaim 16, wherein the intensity modification factor is ten in the powerof the twentieth of the ratio of average empirical decibels per octavemultiplied by the extent of pitch modification expressed in octaves. 18.The computer program product of claim 13, wherein the calculating of theintensity modification factor for each of said frames comprises:calculating a reference value of the intensity corresponding to anoriginal pitch frequency of the original pitch contour for said eachframe, by applying the pitch-to-intensity transformation to the originalpitch frequency; calculating a reference value of the intensitycorresponding to the target pitch frequency of the target pitch contourfor each of said frames by applying the pitch-to-intensitytransformation to the target pitch frequency; and dividing the referencevalue of the intensity corresponding to the target pitch frequency bythe reference value of the intensity corresponding to the original pitchfrequency.
 19. A system comprising: (i) a non-transitory storage devicehaving stored thereon instructions for: receiving an utterance, anoriginal pitch contour of the utterance, and a target pitch contour forthe utterance, wherein the utterance comprises a plurality ofconsecutive frames, and wherein at least one of said frames is a voicedframe, calculating the original intensity contour of said utterance,generating a pitch-modified utterance based on the target pitch contour,calculating an intensity modification factor for each of said frames,based on said original pitch contour and on said target pitch contour,to produce a sequence of intensity modification factors corresponding tosaid plurality of said consecutive frames, calculating a final intensitycontour for said utterance by applying said intensity modificationfactors to said original intensity contour, and generating acoherently-modified speech signal by time-dependent scaling of theintensity of said pitch-modified utterance according to said finalintensity contour; and (ii) at least one hardware processor configuredto execute said instructions.
 20. The system of claim 19, furthercomprising a database, wherein: each of said frames is mapped to acorresponding speech class selected from a predefined set of speechclasses, the calculating of the intensity modification factor for eachof said frames is based on a pitch-to-intensity transformationrepresented as a function of a pitch frequency and a set of controlparameters, and the database comprises values of said control parametersper a speech class of said set of speech classes, and wherein saidstorage device have further stored thereon instructions for settingvalues for said control parameters for each of said frames according toits corresponding speech class, wherein the values are fetched from saiddatabase.